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ABSTRACT 

The  Defence  R&D  Canada  -  Valcartier  has  undertaken  R&D  work  to  provide  Canadian  warships  with  a 
revolutionary  decision  support  capability  explicitly  designed  to  make  the  automation  transparent  and  provide 
warfighters  with  effective  and  trusted  decision  support.  This  effort  uses  cognitive  system  engineering  (CSE) 
as  the  essential  design  framework.  CSE  approaches  are  well  suited  to  deal  with  decision  support  issues  but 
are  in  practice  very  expensive  to  conduct,  time  consuming  and  more  important,  generally  inefficient  from  a 
design  process  perspective.  With  the  latter  limitations  in  mind,  a  pragmatic  CSE  approach,  known  as  the 
Applied  Cognitive  Work  Analysis  (ACWA),  was  developed  to  bridge,  in  a  structure,  efficient  and  converging 
way,  the  gap  between  cognitive  analysis  and  design.  As  a  result,  the  cost  to  conduct  CSE  analyses  using  the 
ACWA  approach  is  reduced  and  the  analysis -design  efficiency  is  significantly  improved,  therefore  making 
easier  the  identification  of  decision-aiding  concepts  suited  to  provide  effective  decision  support. 

This  paper  presents  a  brief  overview  of  CSE  analyses  methods  and  their  benefits.  The  paper  will  describe  in 
more  detail  the  actual  CSE  approach,  known  as  the  Applied  Cognitive  Work  Analysis  (ACWAf ,  used  in 
decision  support  R&D  investigations.  Finally,  the  paper  presents  some  lessons  learned  when  using  ACWA  for 
a  specific  project. 


1.0  INTRODUCTION 

Command  and  control  (C2)  can  be  associated  to  a  dynamic  human  decision  making  process.  In  military 
environments  known  to  be  non-collaborative,  the  C2  process  is  stressed  mainly  by  real-time  and  uncertainty 
issues.  Indeed,  the  technological  evolution  constantly  increases  the  lethality  and  the  reach  of  the  weapons, 
the  scope  of  the  battlefield  and  the  tempo  of  the  engagement.  Successful  exploitation  of  tactical  resources 
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(e.g.:  sensors,  weapons)  during  the  eonduet  of  C2  aetivities  is  decision  dependent  and  therefore  is  linked  with 
the  modelling  and  design  of  the  C2  algorithms  to  support  the  deeisions  and/or  eognitive  demands  in  a  timely 
manner  using  uneertain  information.  Deeision  support  is  thus  required  to  eope  with  human  limitations  in  sueh 
environments.  This  emphasizes  the  need  for  real-time,  eomputer-based  Deeision  Support  Systems  (DSS) 
in  order  to  bridge  the  gap  between  the  eognitive  demands  inherent  to  the  aeeomplishment  of  the  C2  proeess 
and  the  human  limitations. 

Typieally,  a  technologieal  perspeetive  of  C2  has  led  system  designers  to  prescribe  decision  support  solutions 
to  overcome  many  of  the  domain  problems  but  without  knowing  explicitly  the  decisions  and  cognitive 
demands  that  needed  to  be  supported.  This  and  the  lack  of  knowledge  in  CSE  have  in  the  past  jeopardised  the 
design  of  helpful  computerized  aids  aimed  at  complementing  and  supporting  human  cognitive  tasks. 
Moreover,  this  lack  of  knowledge  has  most  of  the  time  created  new  problems  in  trusting  the  supporting  tools 
and  human  in  the  loop  concerns.  Therefore,  formal  DSS  design  approaches,  such  as  cognitive  engineering 
analyses,  are  required  to  understand  the  problem  before  prescribing  any  automation  solutions.  The  Defence 
R&D  Canada  -  Valcartier  has  undertaken  R&D  work  to  provide  Canadian  warships  with  a  revolutionary  DSS 
in  support  of  their  C2  (e.g.:  tracking,  identification,  situation  and  threat  assessment  and  tactical  resources 
management)  in  the  context  of  above  water  warfare.  This  effort  uses  one  specific  CSE  approach  as  the 
essential  design  framework. 

This  paper  presents  a  brief  overview  of  CSE  analyses  methods  and  their  benefits.  This  overview  is  done  in 
light  of  a  feasibility  study  done  at  DRDC  Valcartier  to  investigate  the  applicability  of  Cognitive  Work 
Analysis  (CWA)  to  model  decision  support  in  command  and  control.  The  paper  will  describe  in  more  detail 
the  actual  CSE  approach,  known  as  the  Applied  Cognitive  Work  Analysis  (ACWA),  used  in  support  ongoing 
decision  support  R&D  investigations.  This  ACWA  modeling  method  is  a  pragmatic  adaptation  of  the  CWA 
method  in  order  to  cope  the  limitations  related  to  applying  CWA.  Finally,  the  paper  presents  some  lessons 
learned  when  using  ACWA  for  a  specific  case  study. 


2.0  COGNITIVE  SYSTEM  ENGINEERING  ANALYSES 

CSE  analyses  are  defined  as  approaches  that  aim  to  develop  knowledge  about  the  interaction  between  human 
information  processing  capacities  and  limitations,  and  technological  information  processing  systems. 
The  usefulness  of  a  system  is  closely  related  to  its  compatibility  with  the  human  information  processing. 
Therefore,  CSE  analyses  focus  on  the  cognitive  demands  imposed  by  the  world  to  specify  how  technology 
should  be  exploited  to  reveal  the  problems  intuitively  to  the  decision  maker’s  brain. 

2,1  Cognitive  Work  Analysis 

Among  the  procedures  developed  to  identify  cognitive  processes,  there  are  the  Cognitive  Task  Analysis 
(CTA)  and  the  Cognitive  Work  Analysis  (CWA).  There  are  only  subtle  and  ambiguous  differences  between 
these  two  procedures.  Moreover,  their  labels  are  frequently  used  in  an  interchangeable  manner  in  the 
literature.  However,  the  CWA  can  be  seen  as  a  broader  analysis  than  the  CTA.  According  to  Vicente  [Ref  1], 
traditional  task  analysis  methods  typically  result  in  a  single  temporal  sequence  of  overt  behaviour.  This 
description  represents  the  normative  way  to  perform  the  task.  However,  traditional  methods  cannot  account 
for  factors  like  changes  in  initial  conditions,  unpredictable  disturbances  and  the  use  of  multiple  strategies. 
The  use  of  the  traditional  task  analysis  brings  an  artifact  that  will  only  support  one  way  to  perform  the  task. 

Vicente  proposes  an  ecological  approach  in  which  the  three  factors  above  are  considered.  The  ecological 
approach,  which  can  be  seen  as  a  CWA,  takes  its  origin  in  psychological  theories  that  were  first  advanced  by 
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Brunswick  [Ref.  2]  and  Gibson  [Refs. 3-4].  These  researchers  raised  the  importance  to  study  the  interaction 
between  the  human  organism  and  its  environment.  The  perception  of  an  object  in  the  environment  is  a  direct 
process,  in  which  information  is  simply  detected  rather  than  being  constructed  [Ref  4].  The  human  and  the 
environment  are  coupled  and  cannot  be  studied  in  isolation.  A  central  concept  of  this  approach  is  the  notion  of 
affordance.  The  affordance  is  an  aspect  of  an  object  that  makes  it  obvious  how  the  object  is  to  be  used. 
Examples  are  a  panel  on  a  door  to  indicate,  “push”,  and  a  vertical  handle  to  indicate  “pull”  [Ref  5].  When  the 
affordance  of  an  object  is  obvious,  it  is  easy  to  know  how  to  interact  with  it.  The  environment  in  which  a  task 
is  performed  has  a  direct  influence  in  the  overt  behaviour.  Hence,  the  ecological  approach  begins  by  studying 
the  constraints  in  the  environment  that  are  relevant  to  the  operator.  These  constraints  influence  the  observed 
behaviour. 

The  ecological  approach  [Ref  6]  is  comparable  to  and  compatible  with  Rasmussen’s  abstraction  hierarchy 
framework  [Refs  7-8].  Rasmussen’s  framework  is  used  for  describing  the  functional  landscape  in  which 
behaviour  takes  place  in  a  goal-relevant  manner.  This  abstraction  hierarchy  is  represented  by  means-ends 
relations  and  is  structured  in  several  levels  of  abstraction  that  represent  functional  relationships  between  the 
work  domain  elements  and  their  purposes.  With  the  ecological  approach,  Rasmussen  has  developed  a 
comprehensive  methodology  for  CWA  that  overcomes  the  limitations  of  traditional  CTA  by  taking  into 
account  the  variability  of  performance  in  real-life,  complex  work  domains. 

Conducting  CWA  requires  conducting  sequentially  five  different  type  of  analysis.  As  indicated  in  Table  1, 
findings  from  each  analysis  activity  provide  a  specific  type  of  design  information  that  are  captured  using  a 
specific  modelling  tool. 


Table  1:  CWA  Phases 


Phases  ofCWA 

Kinds  of  Information 

Modeling  Tools 

Work  Domain  Analysis 

Purpose  and  structure  of 
work  domain 

Abstraction-decomposition 

space 

Control  Task  Analysis 

Goals  to  be  satisfied, 
decisions/cognitive 
processing  req'd 

Decision  ladder  templates 

Strategies  Analysis 

Ways  that  control  tasks 
can  be  executed 

Infomiation  Flow  Maps 

Social  Organisation  and 
Cooperation  Analysis 

Who  carries  out  work  and 

how  it  is  shared 

Armotations  on  all  the  above 

Competencies  Analysis 

Kinds  of  mental  processing 
supported 

Skills,  Rules  and  Knowledge 
models 

CWA  seems  to  be  the  best  choice  to  answer  questions  related  to  understanding  the  C2  task.  Recently, 
a  feasibility  study  to  investigate  the  applicability  of  CWA  for  C2  was  performed  to  confirm  this  [Ref  9]. 
The  study  revealed  that  the  methodology  is  well  suited  to  deal  with  decision  support  design  issues  but  is  in 
practice,  if  done  in  a  full  scale  (all  sequential  CWA  phases)  for  a  small  problem,  time  consuming  and  very 
expensive  to  conduct,  and  the  quality  of  the  findings  is  dependent  on  subject  matter  experts’  availability  and 
on  the  skills  of  the  people  conducting  it.  In  addition,  the  study  did  not  show  the  gap  between  cognitive 
analyses  and  design  making  the  DSS  engineering  process  inefficient. 
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3.0  APPLIED  COGNITIVE  WORK  ANALYSIS 

The  ACWA  methodology  [Ref.  10]  emphasizes  a  stepwise  process  to  reduce  the  gap  to  a  sequence  of  small, 
logical  engineering  steps... each  readily  achievable.  At  each  intermediate  point  the  resulting  decision-centered 
artifacts  create  the  spans  of  a  design  bridge  that  link  the  demands  of  the  domain  as  revealed  by  the  cognitive 
analysis  to  the  elements  of  the  decision  aid. 

The  ACWA  approach  is  a  structured,  principled  methodology  to  systematically  transform  the  problem  from  an 
analysis  of  the  demands  of  a  domain  to  identifying  visualizations  and  decision-aiding  concepts  that  will 
provide  effective  support.  The  steps  in  this  process  include: 

•  Using  a  Functional  Abstraction  Network  (FAN)  model  to  capture  the  essential  domain  concepts  and 
relationships  that  define  the  problem-space  confronting  the  domain  practitioners; 

•  Overlaying  Cognitive  Work  Requirements  (CWR)  on  the  functional  model  as  a  way  of  identifying  the 
cognitive  demands  /  tasks  /  decisions  that  arise  in  the  domain  and  require  support; 

•  Identifying  the  Information  /  Relationship  Requirements  (IRR)  for  successful  execution  of  these 
cognitive  work  requirements; 

•  Specifying  the  Representation  Design  Requirements  (RDR)  to  define  the  shaping  and  processing  for 
how  the  information  /  relationships  should  be  represented  to  practitioner(s); 

•  Developing  Presentation  Design  Concepts  (PDC)  to  explore  techniques  to  implement  these 
representation  requirements  into  the  syntax  and  dynamics  of  presentation  forms  in  order  to  produce 
the  information  transfer  to  the  practitioner(s). 

In  the  ACWA  analysis  and  design  approach,  design  artifacts  are  created  to  capture  the  results  of  each  of  these 
intermediate  stages  in  the  design  process.  These  design  artifacts  form  a  continuous  design  thread  that  provides 
a  principled,  traceable  link  from  cognitive  analysis  to  design.  However,  the  design  progress  occurs  in  the 
thought  and  work  in  accomplishing  each  step  of  the  process;  by  the  process  of  generating  these  artifacts. 
The  artifacts  serve  as  a  post  hoc  mechanism  to  record  the  results  of  the  design  thinking  and  as  stepping  stones 
for  the  subsequent  step  of  the  process.  Each  intermediate  artifact  also  provides  an  opportunity  to  evaluate  the 
completeness  and  quality  of  the  analysis/design  effort,  enabling  modifications  to  be  made  early  in  the  process. 
The  linkage  between  artifacts  also  ensures  an  integrative  process;  changes  in  one-artifact  cascades  along  the 
design  thread  necessitating  changes  to  all.  Figure  1  provides  a  visual  depiction  of  the  sequence  of 
methodological  steps  and  their  associated  output  artifacts,  as  well  as  an  indication  that  the  process  is  typically 
repeated  in  several  expanding  spirals,  each  resulting  in  an  improved  Decision  Support  System.  The  following 
subsections  describe  briefly  the  ACWA  steps  but  the  reader  should  read  Reference  10  for  a  more  complete 
and  detailed  description  of  the  methodology. 
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Information  /  Relationship  Requirements: 

Defining  What  Content  is  Needed 
for  Effective  Decision-Making 


Supporting  Information 

Needs 


Dib.'t  Location  of  combat  resources  cjrrentlywnicn  have 
combat  power  ranges/effectiveness  that  incluae  the 
specified  point  In  time  and  space. 

-Dlb.2  Ttmsreqiiired  tor8-aim-such.coinbal,  if  required . ^ 

D1b.3  Measures  of  combat  power  of  the  combat  resource?' 
currently  within  range  of  the  specified  point  In  time  anp-’ 
space,  (this  one  needs  help) 


Decision  Requirements/ 


Abstracted  Decision:  / 

D1  —  "Choose  combined  combat  power  to  achieve  the 
objectives  at  a  specific  point  In  time  and  space' 

Secondary  Decisions:  / 

Dia- "Monitor  the  enemy’s  state  after  the  ei^my  reaches 
the  specific  point  In  time  and  space.  (Goal;6atisfaction 
Monitoring) A/ ^ 


Dib- "Choose  among  the  combat  resoup:es  thgt  bar 
their  combat  power  to  bear  on  the  specific  pgini  In  tin 
space."  (Planning  -  process  availability)'' ■ 


Ulc  -  'LsLale  Ike  Lnemy's  sUe  aller  Ihe  appl.cal.on  ol 

the  chosen  combat  power  at  the  specific  point  In  space  and 
time."  (Planning  -  choice  among  alternatives) 


Side  Effects  (Other  Impacted  Goals): 

What  will  the  Impact  be  of  moving  combat  power  (a)  to  a 
specific  point  In  space  and  time.  (I.e.,  what  else  were  they 
doing  and  what  will  happen  If  they  change  assignment?) 


Workspace  Design 


User  Context 


Decisions  in  Scope 


r= 


Cognitive  Work 

Requirements: 
Identifying  the  Cognitive 
Demands  of  the  Problem 
Space 


Functional  Abstraction  Network: 
Modeling  Critical  Domain  Relationshi 


Representation 

Design 

Requirements: 
Defining  Relationship 
Between 

Requirements  and 
Visualization  Concept 


Presentation  Design 

Concepts: 

Making  the  Problem 
Transparent 


Figure  1 :  A  sequence  of  analysis  and  design  steps  create  a  continuous  design  thread  that 
starts  with  a  representation  of  domain  concepts  and  relationships  through  development 
of  decision  support  requirements  to  creation  of  visualization  and  aiding  concepts  and 
rapid  prototypes  with  which  to  explore  the  design  concepts. 


3,1  Modeling  the  Work  Domain 

The  work  domain  analysis  is  performed  based  on  a  variety  of  Knowledge  Elicitation  (KE)  activities. 
This  involved  interactions  with  expert  practitioners  in  the  domain  and  included  face-to-face  interviews  with 
the  experts,  watching  the  experts  work  in  the  domain,  and  observations  in  simulated  exercises  with  scenarios 
crafted  to  address  specific  aspects  of  the  work  domain.  In  practice,  this  is  an  iterative,  progressively  deepening 
process.  The  key  is  to  focus  on  progressively  evolving  and  enriching  the  model  so  as  to  ultimately  discover  an 
understanding  of  the  goal-driven  characteristics  of  the  domain  that  will  lead  to  an  understanding  of  the 
decisions  practitioners  are  faced  with  in  the  domain. 

The  phrase  ‘bootstrapping  process’  has  been  used  to  describe  this  process  and  emphasize  the  fact  that  the 
process  builds  on  itself  [Ref  11].  Each  step  taken  expands  the  base  of  knowledge  providing  opportunity  to 
take  the  next  step.  Making  progress  on  one  line  of  inquiry  (understanding  one  aspect  of  the  field  of  practice) 
creates  the  room  to  make  progress  on  another.  One  starts  from  an  initial  base  of  knowledge  regarding  the 
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domain  and  how  practitioners  function  within  it  (often  very  limited).  One  then  uses  a  number  of  KE 
techniques  to  expand  on  and  enrich  the  base  understanding  and  evolve  an  ACWA  model  from  which  ideas  for 
improved  support  can  be  generated.  For  example,  one  might  start  by  reading  available  documents  that  provide 
background  on  the  field  of  practice  (e.g.,  training  manuals,  procedures),  the  knowledge  gained  will  raise  new 
questions  or  hypotheses  to  pursue  that  can  then  be  addressed  in  interviews  with  domain  experts,  it  will  also 
provide  the  background  for  interpreting  what  the  experts  say.  In  turn,  the  results  of  interviews  or  exercises 
may  point  to  complicating  factors  in  the  domain  that  need  to  be  modeled  in  more  detail  in  the  Functional 
Abstraction  Network  (FAN).  This  provides  the  necessary  background  to  create  scenarios  to  be  used  to  observe 
practitioner  performance  under  simulated  conditions  or  to  look  for  confirming  example  cases  or  interpret 
observations  in  naturalistic  field  studies. 

The  FAN  provides  a  framework  for  making  explicit  the  goals  to  be  achieved  in  the  domain  and  the  alternative 
means  available  for  achieving  those  goals.  High-level  goals,  such  as  impacting  a  critical  function, 
are  decomposed  into  supporting  lower-level  subgoals.  This  provides  the  basis  for  identifying  -  through 
subsequent  steps  in  the  analysis  and  design  process  -  the  cognitive  activities  that  arise  in  the  domain  and  the 
information  needed  to  support  those  decisions.  The  FAN  enables  the  designer  to  determine  where  decision¬ 
making  is  likely  to  be  difficult  due  to  the  fundamental  characteristics  of  the  domain.  For  example,  the  FAN 
helps  convey  places  in  the  problem  space  where  objectives  compete  with  each  other  (e.g.,  where  choices  have 
to  be  made  that  require  some  level  of  sacrificing  of  one  objective  in  order  to  achieve  another,  perhaps  more 
heavily  weighted,  objective),  or  otherwise  constrain  each  other  (e.g.,  where  the  satisfaction  of  multiple  goals 
need  to  be  considered  in  determining  the  best  course  of  action). 

3,2  Modeling  Cognitive  Demands 

With  the  FAN  representation  of  the  work  domain  as  the  underlying  framework,  it  is  possible  to  derive  the 
cognitive  demands  for  achieving  domain  goals.  In  our  methodology,  we  refer  to  these  demands  as  cognitive 
requirements.  Thus,  the  term  ‘decision’  is  used  in  a  broad  sense.  Based  on  the  underlying  premises  of  the 
modeling  methodology,  these  decisions  center  around  goal-directed  behavior,  such  as  monitoring  for  goal 
satisfaction  and  resource  availability,  planning  and  selection  among  alternative  means  to  achieve  goals, 
and  controlling  activities  (initiating,  tuning,  and  terminating)  to  achieve  goals  [Ref.  12]  as  well  as 
collaboration  activities  in  team  settings  [Ref  13].  By  organizing  the  specification  of  cognitive  requirements 
around  nodes  in  the  goal-means  structure,  rather  than  organizing  requirements  around  predefined  task 
sequences  (as  in  traditional  approaches  to  task  analysis),  the  representation  helps  insure  that  the  resulting 
design  concepts  reflect  a  decision-centered  perspective.  The  resulting  decision  support  concepts  will  thus 
support  domain  practitioners  in  understanding  the  goals  to  be  achieved  and  what  decisions  and  actions  need  to 
be  taken  to  achieve  these  goals. 

The  cognitive  demands  that  are  derived  from  a  cognitive  analysis  of  the  work  domain  constitute  a  second  key 
modeling  artifact  -  Cognitive  Requirements  (CR)s.  CRs  are  tied  directly  to  nodes  in  the  FAN  and  provide  an 
intermediate  artifact  that  forms  the  essential  part  of  the  design  thread,  eventually  providing  an  end-to-end 
connection  from  goal  nodes  in  the  FAN  to  supporting  decision  support  concepts. 

The  FAN  forms  the  basis  for  the  structure  of  the  decision-making  activities  that  will  be  reflected  in  the 
Decision  Requirements.  For  example,  every  Goal  node  in  the  FAN  has  associated  “Goal  Monitoring”  types  of 
decisions.  Fikewise,  Processes  have  associated  “Process  Monitoring”  decisions.  Similarly,  there  will  always 
be  “Feedback  Monitoring”  types  of  decisions  related  to  assessing  whether  actions  are  achieving  desired  results 
as  well  as  monitoring  side  effects  of  actions.  Depending  on  the  relationships  between  nodes  in  the  FAN, 
there  will  be  decisions  related  to  “Control”  (e.g.,  selection  of  alternative  means  to  achieve  a  particular  goal) 
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and  “Abnormality  Detection  /  Attention  Focusing”.  Table  2  contains  an  example  ‘template’  of  these  generic 
CR’s  that  are  used  to  stimulate  the  identification  of  specific  CRs  at  each  of  the  nodes  in  the  FAN: 

Table  2:  Template  of  Typical  Decision  Types 


Generic  Cognitive  /  Decision  Requirements 

Monitoring  /  Situation  Awareness: 

Goaf  Monitoring: 

Feedback  Monitoring: 

Goal  Satisfaction:  Are  the  function-related  goals  satisfied  under  current 
conditions? 

Procedure  adequacy:  Is  the  current  procedure  achieving  the  desired 
goals? 

Margin  to  Dissatisfaction:  Are  goal  limits/restrictions  being  approached? 

Control  action  feedback:  Are  the  operator  control  actions  achieving 
their  desired  goals? 

Process  Monitoring: 

Control: 

Active  processes;  What  processes  are  currently  active?  What  is  the  relative 
contribution  of  each  of  the  active  processes  to  goal  achievement?  Are  the 
processes  performing  correctly? 

Process  control:  How  is  the  process  controlled  for  process 
deployment,  tuning  for  optimum  performance,  termination? 

Process  element  monitoring:  Are  the  individual  processes  and  their 
components  working  as  they  are  supposed  to? 

Manual  take-over:  if  intervention  is  required,  what  actions  must  be 
taken? 

Automation  monitoring:  Are  automated  support  systems  functioning 
properly?  What  goals  are  the  automated  support  systems  attempting  to 
achieve?  Are  these  appropriate  goals? 

Abnormality  Detection /Attention  Focusing: 

Limit  Crossing;  Has  any  goal  or  process  component  exceeded  an 
established  administrative,  procedural,  or  design  limit? 

Procedure  ‘trigger’:  Has  any  goal  or  process  component  reached  a  value 
that  a  procedure  uses  as  a  triggering  event? 

Automatic  system  ‘trigger’:  Has  any  goal  or  process  component  reached  a 
value  that  an  automatic  system  uses  as  an  initiating  value? 

The  key  issue  here  is  that  this  template  is  not  meant  to  be  a  rote,  ‘turn  the  crank’  type  of  process.  Rather,  these 
questions  are  meant  to  be  a  guide  to  stimulate  thinking  about  relevant  decision-making  in  the  context  of  a 
FAN  model  of  the  target  work  domain.  Each  domain  is  unique  in  the  decision-making  demands  imposed  on 
the  human  operators.  As  such,  each  work  domain  will  require  slightly  different  variants  of  these  questions. 
Successful  elucidation  of  decision  requirements  will  also  depend  on  corroboration  from  multiple  data  sources, 
including  case  studies,  interviews,  observations,  etc.  In  addition,  guiding  insights  can  come  from  research 
on  similar  work  domains  as  well  as  basic  research  on  human  cognition,  decision-making,  biases,  and  errors. 
For  example,  previous  work  on  decision  making  in  dynamic,  high-risk  worlds  can  guide  analysis  and 
interpretation  of  analogous  worlds  in  terms  of  potential  points  of  complexity,  typical  decision  making 
difficulties  and  strategies,  and  critical  characteristics  of  difficult  problem-solving  scenarios. 

3,3  Capturing  the  Means  for  Effective  Decision-Making 

The  next  step  in  the  process  is  to  identify  and  document  the  information  required  for  each  decision  to  be 
made.  Information  requirements  are  defined  as  the  set  of  information  elements  necessary  for  successful 
resolution  of  the  associated  decision  requirement.  This  set  of  information  constitutes  the  third  key  modeling 
artifact  -  Information  Requirements  (IR)s.  The  focus  of  this  step  in  the  methodology  is  on  identifying  the  ideal 
and  complete  set  of  information  for  the  associated  decision-making. 
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Information  Requirements  speeify  mueh  more  than  speeifie  data  elements;  it  is  data  in  eontext  that  beeomes 
information  [Refs.  14-15].  The  data- to- information  relationship  ean  be  eomplex  and  require  a  signifieant 
amount  of  eomputations  and/or  transformations.  For  this  reason  ACWA  is  a  design  approaeh  that  has  a  mueh 
deeper  impaet  on  the  entire  DSS  arehiteeture  than  merely  the  look  and  feel  of  the  final  GUI.  For  example, 
in  the  ease  of  a  thermodynamic  system,  an  IR  might  be  ‘flow  coefficient  with  respect  to  appropriate  limits.’ 
This  requires  the  estimation  of  the  parameter  ‘flow  coefficient’  derived  from  model-based  computations  and 
sensor  values  and  the  comparison  of  that  parameter  against  a  limit  referent.  The  degree  of  transformation 
required  can  vary  from  simple  algebra  to  complex,  intelligent  algorithms.  Reference  16  provides  an  example 
of  Information  Requirements  that  could  only  be  satisfied  by  an  advanced  planning  algorithm  and  significant 
data  transformations. 

In  addition,  it  is  important  to  note  that  identifying  Information  Requirements  is  focused  on  satisfying  the 
decision  requirements  and  is  not  limited  by  data  availability  in  the  current  problem-solving  environment. 
In  cases  where  the  required  data  is  not  directly  available  ACWA  provides  a  rationale  for  obtaining  that  data 
(e.g.,  pulling  data  from  a  variety  of  previously  stove-piped  databases,  adding  additional  sensors,  or  creating 
‘synthetic’  values).  This  is  a  critical  change  from  the  typical  role  that  Human  Factors  Engineers  have  had  in 
the  past  (designing  an  interface  after  the  instrumentation  has  been  specified).  Consequently,  this  type  of  an 
approach  is  fundamentally  broader  in  scope  than  other  approaches  to  interface  design  that  do  not  consider  the 
impact  of  Information  Requirements  on  system  architecture  specifications  [Ref  17]. 

The  specific  context  and  concatenation  of  data  to  form  Information  Requirements  depends  on  the  specific 
Cognitive  /  Decision  Requirement  being  satisfied.  The  same  data  elements  can  be  cast  into  Information 
Requirements  in  different  ways  that  support  very  different  decisions. 

Just  as  the  FAN  representation  provided  the  framework  for  the  derivation  of  decision  requirements, 
the  decision  requirements  provide  the  essential  context  for  the  information  requirements  because  they  indicate 
the  factors  (and  thus  information)  that  will  need  to  be  considered  in  making  decisions. 

3,4  Linking  Decision  Requirements  to  Aiding  Concepts  -  Developing  and  Documenting  a 
‘Model  of  Support’  using  the  Representation  Design  Requirements 

The  FAN  and  its  associated  CWR  and  IRR  ‘overlays’  constitute  a  solid  foundation  for  the  development  of  the 
aiding  concepts  to  form  the  Decision  Support  System.  The  design  of  the  Decision  Support  System  occurs  at 
two  levels:  at  a  micro  level  to  ensure  that  each  presentation  element  effortlessly  communicates  its  information 
to  the  user;  and  at  the  macro  level  to  ensure  that  the  overall  collection  of  presentation  design  concepts 
(the  Decision  Support  System  in  a  holistic  sense)  is  organized  in  an  intuitive  way  that  does  not  add  its  own 
“manage  the  Decision  Support  System”  cognitive  burdens  to  those  of  the  domain. 

This  step  in  the  ACWA  process  develops  the  specification  of  the  display  concept  and  how  it  supports  the 
cognitive  tasks  and  is  captured  in  Representation  Design  Requirements  (RDR)  for  the  eventual  development 
of  Presentation  Design  Concepts  (PDC).  The  RDR  defines  the  goals  and  scope  of  the  information 
representation  in  terms  of  the  cognitive  tasks  it  is  intended  to  support  (and  thus  a  defined  target  region 
of  the  FAN).  It  also  provides  a  specification  of  the  supporting  information  required  to  support  the  cognitive 
tasks.  An  RDR  is  another  span  of  the  bridge  that  helps  to  link  the  decisions  within  the  work  domain 
to  the  visualization  and  decision  support  concepts  intended  to  support  those  decisions.  In  many  cases, 
multiple  design  concepts  may  be  generated  that  attempt  to  satisfy  the  RDR’s  requirements.  Typically,  other 
supporting  artifacts  are  generated  at  this  step  in  the  process  as  required  to  specify  such  issues  as  presentation 
real-estate  allocation,  attention  management  (salience)  across  the  information  to  be  presented,  etc. 
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The  RDR  also  represents  a  eritieal  system  design  eonfiguration  management  tool,  eritieal  for  ensuring 
eoverage  of  the  funetional  deeision  spaee  aeross  all  presentations  and  presentation  design  eoneepts.  The  RDR 
begins  the  shift  in  foeus  from  ‘what’  is  to  be  displayed  to  ‘how’,  ineluding  annotations  on  relative  importanee 
that  maps  to  relative  salienee  on  the  visualization,  ete.  A  eomplete  RDR  is  aetually  a  set  of  requirements 
‘documents’,  each  describing  the  requirements  for  the  intended  representation  of  the  IRRs.  It  contains 
descriptions  of  how  all  presentation  mechanisms  of  the  domain  practitioner’s  workspace  are  to  be  coordinated, 
how  available  audio  coding  mechanisms  are  allocated,  similarly  for  visual,  haptic,  and  any  other  sensory 
channels  to  be  employed.  The  RDR  is  not  only  a  compilation  of  information  developed  earlier,  it  has  the 
added  value  of  a  more  complete  description  of  the  behaviors  and  features  needed  to  communicate  the 
information  effectively  as  well  as  an  allocation  of  the  Information  /  Relationship  Resources  across  the  entire 
set  of  displays  within  the  workspace.  When  done  correctly  it  is  still  in  the  form  of  a  ‘requirement’  and  not  an 
implementation.  This  artifact  becomes  a  key  transition  between  the  Cognitive  System  Engineer,  the  System 
Developer,  and  the  System  (Effectiveness)  Tester. 

The  RDR  also  provides  one  important  ancillary  benefit,  as  long  as  the  domain  remains  unchanged,  the  RDR 
serves  as  an  explicit  documentation  of  the  intent  of  the  presentation  independent  of  the  technologies  available 
and  used  to  implement  the  Decision  Support  System.  As  newer  technologies  become  available,  as  their 
interaction  with  human  perception  become  better  understood,  the  technologies  used  to  implement  the  RDR 
requirements  can  evolve. 

3,5  Developing  Presentations  -  Instantiating  the  Aiding  Concept  as  Presentation 
Design  Concepts 

From  the  RDR’s  specification  of  how  information  is  to  be  represented  within  the  decision  support  system,  the 
next  step  of  the  ACWA  process  is  the  explicit  design  of  Presentation  Design  Concepts  (PDCs)  for  the 
Decision  Support  System.  (A  similar  process  is  used  for  the  design  of  auditory,  visual,  or  other  senses’ 
presentations  of  the  RDR’s  specification.)  This  final  step  requires  an  understanding  of  human  perception  and 
its  interaction  with  the  various  presentation  techniques  and  attributes.  As  such,  it  requires  considerable  skill 
and  ability  beyond  cognitive  work  analysis.  The  actual  design  of  a  revolutionary  aiding  concept  is  probably 
one  of  the  largest  ‘design  gaps’  that  is  needed  to  be  bridged  within  the  ACWA  process.  The  ACWA  design 
practitioner  must  be  fluent  in  the  various  presentation  dimensions:  color,  layout,  line  interactions,  shape, 
edge  detection,  etc.  Essentially  the  designer  must  really  understand  what  characteristics  of  presentation 
implicitly  specify  about  the  interaction  with  the  user’s  perception.  The  conversion  of  the  requirements  in  the 
RDR  to  a  sensory  presentation  form  in  a  PDC  requires  considerable  skill  and  background  in  these  areas. 
With  the  RDR  as  a  guide,  the  sketches,  proposals,  brainstorming  concepts  can  all  be  resolved  back  against  the 
display’s  intent  and  requirements.  The  issues  of  how  it  is  perceived  can  best  be  done  with  empirical  testing  of 
prototypes,  and  often  requires  considerable  tuning  and  adjustment  to  achieve  the  representational  capabilities 
specified  in  the  RDR. 

Of  all  the  steps  in  ACWA,  this  final  presentation  development  requires  a  significant  background  in 
presentation  technologies,  human  perceptual  characteristics,  and  how  they  interact.  The  other  ACWA 
artifacts,  notably  the  RDR,  do  provide  a  test  basis  to  iterate  the  presentation  design  concepts.  By  testing  each 
proposed  display  prototype  against  the  single  indicator  question  of  “does  it  support  the  decisions  it  is 
supposed  to  as  defined  by  the  RDR?”  it  is  possible  to  at  least  identify  unsuccessful  attempts  and  continue  to 
design  toward  a  more  successful  one.  This  last  step  across  the  gap  is  often  difficult,  but  the  ACWA 
methodology  has  made  it  a  much  smaller  step,  from  a  much  more  solid  footing  than  would  be  the  case  if 
attempting  to  directly  design  a  presentation  without  its  RDR  precursor. 
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4.0  LESSONS  LEARNED  FROM  THE  ATAC  PROJECT 

The  first  three  steps  of  the  ACWA  approach,  as  described  in  the  previous  section,  were  used  during  the  initial 
phase  of  the  ATAC  (Advanced  Threat  Assessment  Capability)  project  for  the  Canadian  Navy.  One  important 
requirement  for  that  project  was  to  use  a  formal  modeling  and  design  approach  that  allowed  one  to  trace  back 
the  origin  of  a  specific  algorithm  development.  The  ACWA  approach  in  concert  with  a  systematic 
documentation  methodology  (FAH  and  its  artifacts)  was  found  very  useful  in  the  ATAC  project  as  it  provided 
the  means  to  meet  that  traceability  requirement. 

The  documentation  was  iteratively  revised  and  expanded  and  served  as  the  main  reference  material  to 
conduct/structure  interviews  or  training  session  with  subject  matter  experts.  The  documentation  was  to  also 
used  during  brainstorming  design  meetings  and  helpful  for  progress  review  meetings. 

The  ACWA  was  found  to  be  opportunistic  and  flexible  when  new  knowledge  elicitation  activities  arise  and 
when  the  scope  of  the  project  itself  expanded  significantly.  The  important  thing  to  note  here  is  that  the 
modelling  and  design  work  and  documentation  did  not  collapse  when  these  scope  expansions  occurred  but  in 
fact  adapted  very  well. 

At  the  end  of  a  three  days  meeting  with  naval  officers,  where  the  objective  was  to  validate  the  modelling 
effort  done  using  the  ACWA  approach  during  the  ATAC  project,  the  subject  matter  experts  acknowledged  the 
value  of  the  work  and  characterized  the  resulting  FAFI  and  artifacts  as  “brilliant  and  robust”  with  respect  to 
the  captured  naval  C2  domain  knowledge. 

The  next  phases  of  the  ATAC  project  are  the  derivation  of  system  design  concept  specifications  (including  the 
last  two  steps  of  the  ACWA  approach)  and  the  development  of  the  capability  based  on  the  resulting  system 
design  concept  specifications.  Even  though  the  ACWA  approach  was  found  useful  in  the  initial  phases  of  the 
ATAC  project  to  understand  the  problem  before  prescribing  any  automation  solutions,  no  C2  DSS  has  yet 
been  developed  and  therefore  the  complete  value  of  the  ACWA  method  for  C2  related  problems  is  still  to  be 
demonstrated.  On  the  other  hand,  this  type  of  demonstration/evaluation  is  not  trivial  to  do. 

While  work  continues  at  the  Defence  R&D  Canada  -  Valcartier  toward  the  design  of  DSSs,  an  even  greater 
challenge  is  the  evaluation  of  the  dynamic  human  decision-making  performance  when  using  a  newly  designed 
computer-based  decision  support  capability.  Such  assessment  must  be  focused  on  the  net  (human-system 
combination)  decision-making  effectiveness  and  allow  to  answer  questions  like: 

•  Does  this  new  C2  decision  aid  capability  support  the  decision-maker’s  cognitive  demands  during  his 
C2  activities? 

•  Does  this  new  C2  aid  capability  provide  the  right  set  of  information  in  support  of  a  specific  cognitive 
demand? 

•  Does  this  new  C2  aid  capability  increase  the  net  (human-system  combination)  decision-making 
effectiveness? 

Current  evaluation  methods  are  immature  and  focus  on  system  technical  attributes  or  crude  memory  tests. 
This  is  clearly  not  enough. 
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5.0  CONCLUSION 

In  light  of  the  military  C2  process  and  its  domain  constraints,  this  paper  presented  a  brief  overview  of  CSE 
analyses  methods  and  their  benefits.  This  overview  described  CSE  analyses  as  approaches  that  focuses  on  the 
cognitive  demands  imposed  by  the  world  to  specify  how  technology  should  be  exploited  to  reveal  the 
problems  intuitively  (affordance  concept)  to  the  decision  maker’s  brain.  The  paper  also  described  CSE 
methods  as  well  suited  to  deal  with  decision  support  issues  but  are  in  practice  very  expensive  to  conduct  and 
more  important  inefficient  from  a  design  process  perspective. 

One  specific  CSE  approach,  known  as  the  Applied  Cognitive  Work  Analysis  (ACWA),  The  paper  presented 
in  more  details  used  in  support  of  ongoing  decision  support  R&D  investigations.  The  ACWA  methodology 
emphasizes  a  stepwise  process  to  reduce  the  gap  to  a  sequence  of  small,  logical  engineering  steps... each 
readily  achievable.  At  each  intermediate  point  the  resulting  decision-centered  artifacts  create  the  spans  of  a 
design  bridge  that  link  the  demands  of  the  domain  as  revealed  by  the  cognitive  analysis  to  the  elements  of  the 
decision  aid.  As  a  result,  the  cost  to  conduct  CSE  analyses  using  the  ACWA  approach  is  reduced  significantly 
and  the  analysis-design  efficiency  is  drastically  improved,  therefore  making  easier  the  identification  of 
decision-aiding  concepts  suited  to  provide  effective  decision  support. 

Finally,  the  paper  presented  some  lessons  learned  when  using  ACWA  for  a  specific  case  study.  Finally, 
ACWA  lessons  learned  while  used  during  a  specific  project  was  discussed  in  this  paper  along  with  issues 
related  to  the  performance  evaluation  of  dynamic  human  decision-making  DSS. 
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